Augmented reality automotive accessory customer collaborative design and display

ABSTRACT

A system that allows customers to collaborate with vehicle accessory manufacturers to design accessories in an augmented reality environment. A customer decision making and purchasing experience is greatly improved when viewing a physical showroom model (or other real-world model) of a vehicle they intend to purchase and/or customize. The system allows the user to select and customize virtual accessories for the physical vehicle, then overlay those virtual accessories atop the physical vehicle in an augmented reality environment.

PRIORITY

The application is a non-provisional of and claims priority to U.S.Provisional Application No. 63/125,196, entitled “SYSTEMS AND METHODSFOR ENABLING PRECISE OBJECT INTERACTION WITHIN AN AUGMENTED REALITYENVIRONMENT,” filed on Dec. 14, 2020, naming the same inventors, thedisclosure of which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to systems and methods ofaugmented reality and, more particularly, to methods and systems for thecollaborative design and display of vehicle accessories in an augmentedreality environment.

BACKGROUND

Augmented reality (AR) systems provide a means for virtual objects to bepresented in a real-world (physical) environment when the real-worldenvironment is viewed through a device capable of capturing image dataof the real-world environment and displaying the real-world environmenton a user display device. Virtual objects presented in the real-worldenvironment may be overlays presented to a user or as virtual objectspresented to appear as real-world objects within the environment.

In the automotive industry, customers often desire to customize theirvehicles with various accessories. However, typically these customersare forced to view the accessories in magazines or other images and areunable to view the accessory on the real-world vehicle. As a result,customers are often forced to make substantial investments inaccessories without really knowing if they like the look on theaccessories on their vehicles. In addition, those customers often lackthe ability to customize the accessories as they desire. Ultimately,these shortcomings lead to increased customer dissatisfaction with thevehicle buying experience.

SUMMARY

The present disclosure provides systems and methods for collaborativecustomization of vehicle accessories in an augmented realityenvironment. A virtual automotive accessory is first obtained from auser via a graphical user interface. The virtual accessory correspondsto a physical real-world vehicle being viewed through the user device.In response to the user input, the system customizes the automotiveaccessory through customization of various characteristics of theaccessory. Once customization is complete, the finalized design isdisplayed to the user in an augmented reality environment such that thephysical vehicle appears to incorporate the customized automotiveaccessory.

A system for displaying vehicle accessories in an augmented realityenvironment is also described herein. The system includes a processor toperform operations comprising obtaining a selection of an automotiveaccessory corresponding to a physical vehicle. Thereafter, in responseto user input, the automotive accessory is customized. The system thendisplays the customized automotive accessory to the user in an augmentedreality environment such that the physical vehicle appears toincorporate the customized automotive accessory.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures.

FIG. 1 schematically depicts an example computing network for providingan augmented reality system, according to one or more embodimentsdescribed and illustrated herein;

FIG. 2 schematically depicts an example electronic device for providingan augmented reality environment to a user, according to one or moreembodiments described and illustrated herein;

FIG. 3 depicts an example display providing an illustrative graphicaluser interface menu, according to one or more embodiments described andillustrated herein;

FIG. 4 depicts an illustrative instruction screen on the display of thedevice for identifying an object within the environment, according toone or more embodiments described and illustrated herein;

FIG. 5 depicts an illustrative confirmation menu on the display of thedevice for confirming identification of the object within theenvironment, according to one or more embodiments described andillustrated herein;

FIG. 6 depicts an example overlay of a registration feature for aligningwith the real-world feature on the identified object in the real-worldenvironment, according to one or more embodiments described andillustrated herein;

FIG. 7 depicts an illustrative example of the registration feature beingaligned with the real-world feature on the identified object asdisplayed on the display of the device as a user orients the device toalign the registration feature and the real-world feature, according toone or more embodiments described and illustrated herein;

FIG. 8 depicts an illustrative example of a multi-dimensional wireframemodel of the identified object being registered with the identifiedobject within the real-world environment as captured by the imagingdevice of the device, according to one or more embodiments described andillustrated herein;

FIG. 9 depicts another illustrative view of the multi-dimensionalwireframe model being registered with the identified object within thereal-world environment as additional points of view of the identifiedobject are captured by the imaging device of the device, according toone or more embodiments described and illustrated herein;

FIG. 10 depicts another illustrative view of the multi-dimensionalwireframe model being registered with the identified object within thereal-world environment as additional points of view of the identifiedobject are captured by the imaging device of the device, according toone or more embodiments described and illustrated herein;

FIG. 11 depicts an example virtual accessory corresponding to theidentified object being presented as a seamless component added to theidentified object within the real-world environment on the display ofthe device within an augmented reality environment, according to one ormore embodiments described and illustrated herein;

FIG. 12 depicts an example quote for the object and the selectedaccessories depicted in the augmented reality environment presented onthe display of the device, according to one or more embodimentsdescribed and illustrated herein;

FIG. 13 depicts an illustrative flow diagram of a method forfacilitating an accurate, precise, and seamless presentation of avirtual accessory added to the identified object within the real-worldenvironment on the display of the device in the augmented realityenvironment, according to one or more embodiments described andillustrated herein.

FIG. 14 is a block diagram of a system and flow diagram for thecollaborative design and customization of various accessories ofvehicles, according to certain illustrative embodiments of the presentdisclosure; and

FIG. 15 is a flow chart of a method for customizing and displayingaccessories in an augmented reality, according to certain illustrativemethods of the present invention.

DETAILED DESCRIPTION

Illustrative embodiments and related methods of the present disclosureare described below as they might be employed in a system and method forcollaborative design and display of vehicle accessories in an augmentedreality environment. In the interest of clarity, not all features of anactual implementation or methodology are described in thisspecification. It will of course be appreciated that in the developmentof any such actual embodiment, numerous implementation-specificdecisions must be made to achieve the developers' specific goals, suchas compliance with system-related and business-related constraints,which will vary from one implementation to another. Moreover, it will beappreciated that such a development effort might be complex andtime-consuming but would nevertheless be a routine undertaking for thoseof ordinary skill in the art having the benefit of this disclosure.Further aspects and advantages of the various embodiments and relatedmethodologies of the disclosure will become apparent from considerationof the following description and drawings.

As will be described below, illustrative methods and embodiments of thepresent disclosure allow customers to collaborate with vehicle accessorymanufacturers to design and display the accessories in an augmentedreality environment. Embodiments described herein are generally relatedto improving a customer's decision making and purchasing experience whenviewing a physical showroom model (or other real-world model) of avehicle they intend to purchase and/or customize. For example,dealerships generally have only a few vehicles available for viewing ina showroom. However, there are generally many custom options and/oraccessories that may be included on the vehicle to meet a customer'sdesired vehicle.

Typically, the accessory may be depicted on a computer in a fixed numberof views so that the customer may get a sense for what the accessorywould look like on their vehicle. However, this has several limitations.For example, the customer may wish to view multiple accessories togetherand get a better sense for how the one or more accessories appear whenviewed from many angles which are not available on a small graphicalmodel provided on a computer or in a brochure. Additionally, the usermay want to switch between several options to find the one that bestfits their taste. Further, tension can arise in the decision-makingprocess because customers can be confronted with making decision toinclude an accessory that costs hundreds or thousands of dollars withoutreally having a sense for what it would look like when installed ontheir vehicle.

Accordingly, embodiments of the present disclosure are directed toreducing or eliminating this tension by providing an augmented realitysystem that a user may deploy in a showroom or other location whileviewing image data of a real vehicle through an electronic device andvirtually customizing the vehicle with accessories while moving aroundthe vehicle to view the virtualized accessory. The augmented realitysystem allows a user to select and customize the accessory, and thenvirtually install the accessory onto a real-world vehicle so thecustomer can see how the installed accessory looks and whether theywould like to further customize the feature.

The system described herein also delivers a realistic experience to theuser. The virtualized accessories must appear as realistic as possibleand seamlessly integrate with the image data of the real-world vehicle.In general, graphic rendering for an augmented reality environment thatfuses real-world image data with virtual features requires large amountsof graphics processing resources. However, these resources may not beavailable on conventional systems (e.g., handheld electronic devices)such as a smartphone that the user can practically bring into ashowroom. Therefore, there is a need to provide a system and method thatcan deliver a comparable or better augmented reality experience on ahandheld electronic device. Such a system and method are describedherein.

Embodiments of the present disclosure provide an accurate, precise, andseamless presentation of one or more virtual accessories attached to anobject identified within a real-world environment. In particular, theembodiments described herein provide systems and methods that identify areal-world object in an environment captured by an imaging device andregister the identified object with a multi-dimensional wireframe modelof the object so that virtualized accessories may be seamlesslyassociated with, or “locked onto,” the real-world object in an augmentedreality environment presented to a user on a display. Methods ofidentifying the real-world object and registering the identified objectwith the multi-dimensional wireframe model of the object will bedescribed in detail herein.

It should be understood that, although the present disclosure describesthe systems and methods for providing virtual accessories attached tophysical vehicles, the technology described may be utilized in othercontexts. The technology, for example, without limitation may beutilized in generating augmented reality environments for home buildingand/or renovation, clothing fitting, modeling and purchasing, and thelike.

Embodiments described herein utilize a device such as a smartphone,laptop, tablet or other handheld computing device that includes animaging device, a processor, memory, and a display. The imaging devicemay capture image data of an environment. The image data may be analyzedto identify one or more real-world objects within the environment.Moreover, to generate an augmented reality environment, the image datacaptured by the imaging device may be presented in real-time or nearreal-time on the display of the device. In some instances, the displaymay be the display of the device or may be a virtual reality headset orglasses.

The illustrative embodiments disclosed herein also describe systems andmethods that utilize a trained machine learning model to identifyreal-world objects in an environment captured in the image data therebyfacilitating precise interactions between virtual accessories andreal-world objects within an augmented reality environment. Morespecifically, the augmented reality system described herein may utilizea trained machine learning model such as a neural network or other formof artificial intelligence system to identify a real-world object fromimage data captured by an imaging device.

As used herein, identifying a real-world object may refer to both thedetection and classification of an object within the image data.Furthermore, a machine learning model may be implemented duringregistration of the multi-dimensional wireframe model of the identifiedobject and the identified object captured in the image data beingcaptured by the imaging device. For example, the machine learning modelmay be trained to identify whole objects and isolated features of anobject so that registration of the isolated features of the object andthe multidimensional wireframe model may be accurately and preciselymapped to each other. The multidimensional wireframe model may be apredefined computer aided model of the object. For example, themulti-dimensional wireframe model may be a three-dimensional spatialmodel that may appear as overlaying portions of an image or a live videostream of an object (e.g., a vehicle) within a physical environment.Moreover, the augmented reality system described herein may utilize thetrained machine learning model to associate a virtual object (e.g., avirtual accessory) with a portion of the virtual wireframe model suchthat the virtual object may appear precisely positioned on or integratedwith the object (e.g., a vehicle) within the physical environment.

The following will now describe certain illustrative systems and methodsin more detail with reference to the drawings and where like numbersrefer to like structures. FIG. 1 schematically depicts an examplecomputing network for providing an augmented reality system, accordingto one or more embodiments described and illustrated herein. Asillustrated in FIG. 1 , a network 100 may include a wide area network,such as the internet, a local area network (LAN), a mobilecommunications network, a public service telephone network (PSTN) and/orother network and configured to electronically connect a computingdevice 102, a server 103 for managing content and training machinelearning models, and an electronic device 104 enabled with anapplication for a user to use with a publication. The network 100 may beconfigured to electronically and/or communicatively connect thecomputing device 102, the server 103, and/or the electronic device 104.

The computing device 102 may include a display 102 a, a processing unit102 b and an input device 102 c, each of which may be communicativelycoupled to together and/or to the network 100. The computing device 102may be used to develop multi-dimensional wireframe models for objectssuch as vehicles and virtual accessories. The computing device 102 mayalso be utilized to interface with a server 103 to develop, update,and/or repair machine learning models for identifying object and/ordefining and/or updating registration processes for registering (i.e.,correctly aligning) the multi-dimensional model of an identified objectwith the identified object in the real-world as captured by the imagingdevice. The server 103 may be any computing device configured to storedata, host internet-based applications, configured to be searchable byanother device and/or the like. In some embodiments, the server 103maintains a repository of current multi-dimensional wireframe models forobjects such as a number of makes, models, and trims of vehicles and aplurality of virtual models for accessories of the same. The virtualmodels of accessories may include, for example, virtual models ofrunning boards, wheels, tires, roof racks, grilles, and/or the like.

The electronic device 104 may include one or more components such as animaging device (e.g., a camera), to capture images of one or moreobjects in a physical environment of the user. Additionally, theelectronic device 104 may be configured to access a software applicationwithin which a trained machine learning model may be embedded. Theelectronic device 104 may, via a software application and the network100, retrieve a multi-dimensional wireframe model for use in theaugmented reality environment from the one or more databases associatedwith the server 103, as described in greater detail below. That is, theserver 103 may include one or more databases, within which a pluralityof data related to various characteristics of real world objects may bestored. In embodiments, the computing device 102 the data stored inthese databases to associate a virtual object (e.g., a virtualcomponent) with a portion of the virtual wireframe model such that thevirtual object may appear precisely positioned on or integrated with areal-world object within the augmented reality environment.

Additionally, included in FIG. 1 is the electronic device 104. Theelectronic device 104 may be any mobile or personal computing devicesuch as a laptop, tablet, smartphone, or the like that a user may use tocapture image data of a physical real-world object they desire tocustomize virtually with virtual accessories, for example, in an effortto visualize what the accessory would look like on their vehicle. It isalso understood that while the electronic device 104, the computingdevice 102, and the server 103, are depicted as individual devices,these are merely examples. More specifically, in some embodiments, anytype of computing device (e.g., mobile computing device, personalcomputer, server, and the like) may be utilized for any of thesecomponents. Additionally, while each computing device is illustrated inFIG. 1 as a single piece of hardware, this is also an example.

FIG. 2 schematically depicts an example electronic device 104 forproviding an augmented reality environment to a user, according to oneor more embodiments described and illustrated herein. As illustrated inFIG. 2 , the electronic device 104 may include a processor 230,input/output hardware 231, network interface hardware 232, a camera 233,a motion sensor 234, a display 235, a touch input sensor 236, a datastorage component 238, which optionally stores one or moremulti-dimensional wireframe models and/or virtual accessories (e.g.,models of the virtual accessories) and a memory component 242. Thememory component 242 may be machine readable memory (which may also bereferred to as a non-transitory processor readable memory). The memorycomponent 242 may be configured as volatile and/or nonvolatile memoryand, as such, may include random access memory (including SRAM, DRAM,and/or other types of random access memory), flash memory, registers,compact discs (CD), digital versatile discs (DVD), and/or other types ofstorage components. Additionally, the memory component 242 may beconfigured to store operating logic and/or a machine learning model (MLmodel), each of which may be embodied as a computer program, firmware,or hardware, as an example.

A local interface 246 is also included in FIG. 2 and may be implementedas a bus or other interface to facilitate communication among thecomponents of the electronic device 104. The machine readableinstructions may comprise logic or algorithm(s) written in anyprogramming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or5GL) such as, for example, machine language that may be directlyexecuted by the processor, or assembly language, object-orientedprogramming (OOP), scripting languages, microcode, etc., that may becompiled or assembled into machine readable instructions and stored onthe one or more memory modules 210. In some embodiments, themachine-readable instructions may be written in a hardware descriptionlanguage (HDL), such as logic implemented via either afield-programmable gate array (FPGA) configuration or anapplication-specific integrated circuit (ASIC), or their equivalents.Accordingly, the methods described herein may be implemented in anyconventional computer programming language, as pre-programmed hardwareelements, or as a combination of hardware and software components.

The local interface 246 may be formed from any medium that is capable oftransmitting a signal such as, for example, conductive wires, conductivetraces, optical waveguides, or the like. Moreover, the local interface246 may be formed from a combination of mediums capable of transmittingsignals. In one embodiment, the local interface 246 comprises acombination of conductive traces, conductive wires, connectors, andbuses that cooperate to permit the transmission of electrical datasignals to components such as processors, memories, sensors, inputdevices, output devices, and communication devices. Accordingly, thelocal interface 246 may comprise a vehicle bus, such as for example aLIN bus, a CAN bus, a VAN bus, and the like. Additionally, it is notedthat the term “signal” means a waveform (e.g., electrical, optical,magnetic, mechanical or electromagnetic), such as DC, AC,sinusoidal-wave, triangular-wave, square-wave, vibration, and the like,capable of traveling through a medium. The local interface 246communicatively couples the various components of the electronic device104. As used herein, the term “communicatively coupled” means thatcoupled components are capable of exchanging data signals with oneanother such as, for example, electrical signals via conductive medium,electromagnetic signals via air, optical signals via optical waveguides,and the like.

The processor 230 may include any processing component(s) configured toreceive and execute programming instructions (such as from the datastorage component 238 and/or the memory component 242). The instructionsmay be in the form of a machine-readable instruction set stored in thedata storage component 238 and/or the memory component 242. Theinput/output hardware 231 may include a monitor, keyboard, mouse,printer, microphone, speaker, and/or other device for receiving,sending, and/or presenting data. The network interface hardware 232 mayinclude any wired or wireless networking hardware, such as a modem, LANport, Wi-Fi card, WiMax card, mobile communications hardware, and/orother hardware for communicating with other networks and/or devices.

The processor 230 is communicatively coupled to the other components ofthe electronic device 104 by the local interface 246. Accordingly, thelocal interface 246 may communicatively couple any number of processorswith one another, and allow the modules coupled to the local interface246 to operate in a distributed computing environment. Specifically,each of the modules may operate as a node that may send and/or receivedata.

The camera 233 may be any device having an array of sensing devices(e.g., pixels) capable of detecting radiation in an ultravioletwavelength band, a visible light wavelength band, or an infraredwavelength band. The camera 233 may have any resolution. The camera 233may be an omni-directional camera, or a panoramic camera. In someembodiments, one or more optical components, such as a mirror, fish-eyelens, or any other type of lens may be optically coupled to camera 233.In embodiments, the camera 233 may have a broad angle feature thatenables capturing digital content within a 150 degree to 180-degree arcrange. Alternatively, the camera 233 may have a narrow angle featurethat enables capturing digital content within a narrow arc range, e.g.,60 degrees to 90-degree arc range. In embodiments, the one or morecameras 233 may be capable of capturing high definition images in a720-pixel resolution, a 1080-pixel resolution, and so forth.Alternatively, or additionally, the camera 233 may have thefunctionality to capture a continuous real time video stream for apredetermined time period.

The motion sensor 234 may include any device capable of detectingacceleration changes in the electronic device 104 and/or roll, pitch,and yaw rotations. For example, the motion sensors 234 may include anaccelerometer, a gyroscope, or the like.

The display 235 may include any medium capable of transmitting anoptical output such as, for example, a cathode ray tube, light emittingdiodes, a liquid crystal display, a plasma display, a virtual realityheadset or glasses, or the like. Moreover, the display 235 may be atouchscreen enabled by a touch input sensor 236 that, in addition toproviding optical information, detects the presence and location of atactile input upon a surface of or adjacent to the display 235.

It should be understood that the data storage component 238 may residelocal to and/or remote from the electronic device 104 and may beconfigured to store one or more pieces of data for access by theelectronic device 104 and/or other components. It should also beunderstood that the components illustrated in FIG. 2 are merelyexemplary and are not intended to limit the scope of this disclosure.More specifically, while the components in FIG. 2 are illustrated asresiding within the electronic device 104, this is merely an example. Insome embodiments, one or more of the components may reside external tothe electronic device 104. Similarly, while FIG. 2 is directed to theelectronic device 104, other components such as the computing device 102and the server 103 may include similar hardware, software, and/orfirmware.

The network interface hardware 214 may include a chipset (e.g., antenna,processors, machine readable instructions, etc.) to communicate overwireless computer networks such as, for example, wireless fidelity(Wi-Fi), WiMax, Bluetooth, IrDA, Wireless USB, Z-Wave, ZigBee, or thelike. In some embodiments, the network interface hardware 214 includes aBluetooth transceiver that enables the electronic device 104 to exchangeinformation with the server 103 via Bluetooth communication.

Referring now to FIGS. 3-12 , illustrative walkthrough of the operationof the augmented reality system. FIG. 3 depicts an example home page 302(e.g., a graphical user interface menu) displayed on a display 235 ofthe electronic device 104, according to one or more embodimentsdescribed and illustrated herein. In some embodiments, the example homepage 302 may be displayed on the display 235 of the electronic device104 responsive to a user accessing a software application, e.g., byclicking an icon displayed on a screen of the electronic device 104. Inembodiments, the home page 302 may include a plurality of userselectable icons, e.g., such as “Trucks”, “Cars”, “Minivans”, “SUVs”,and “Crossovers”, which correspond to various vehicle brands, makes,models, and the like. In some embodiments, a user may select aparticular brand of truck, such as Tacoma 2020, and choose an iconlabeled “Continue” to initiate operation of the augmented reality systemdescribed herein. Upon selection of the “Continue” icon, the processor230 may instruct the software application to open an initiation page, asdepicted in FIG. 4 .

FIG. 4 depicts an initiation page displayed on a display 235 of theelectronic device 104, according to one or more embodiments describedand illustrated herein. In embodiments, a user may recognize the type ofa vehicle located within a certain proximity of the user in a physicalenvironment of the user, e.g., on the showroom floor of a car show, onthe floor of a dealership, etc. Thereafter, in some embodiments, theuser may choose the type of vehicle (e.g., a Truck) and select the“continue icon 302” from the example home page 302, as depicted in FIG.3 . In response, the processor 230 may instruct the software applicationto display an initiation page 402 on the display 235 of the electronicdevice 104. In embodiments, the initiation page 402 may include datarelated to various characteristics of the example vehicle 404, e.g.,such as the year, make, and model of the vehicle. Additionally, inembodiments, the initiation page 402 may include a real-time videostream of the example vehicle 404 that is captured in the physicalenvironment of the user. In other words, a live video stream of theexample vehicle 404 as present in the physical environment of the user(e.g., on the floor of a car dealership). The data related to thecharacteristics of the example vehicle 404 may be displayed on theinitiation page 402 adjacent to the live video stream (or an image) ofthe vehicle.

In embodiments, the initiation page may instruct a user to perform oneor more actions relative to the example vehicle 404, e.g., place thecamera 233 in a particular position, at a particular angle, and/or at aparticular distance relative to the example vehicle 404.

The electronic device 104 may receive from the camera 233 image data ofthe environment including the vehicle 404. The electronic device 104then processes the image data to identify a real-world object in theenvironment captured by the camera. For example, the electronic device104 may implement a machine learning model to analyze the image data anddetermine the classification such as the brand, make, model, and or trimof the vehicle in the image data. The machine learning model may betrained using training images that include a variety of informationrelated to different vehicles, e.g., colors, makes, models, size ofvehicle wheels, vehicle brand logos, decals, number of doors in avehicle, shapes, contours, and dimensions of vehicles, etc.Additionally, these training images may have been captured underdifferent lighting and environmental conditions. These images may alsoinclude numerous labels corresponding to each of these vehiclecharacteristics. In embodiments, the training images may be input into asupervised machine learning environment that may include a supervisedtraining interface in which an individual may add, delete, or modifythese labels.

Additionally, in certain illustrative embodiments, upon completion ofthe training, the machine learning model may be configured to identify,in real time, one or more characteristics of a particular vehicle (e.g.,dimensions, contours, etc.) from analyzing one or more perspectiveimages of the vehicle that may be captured in the physical environmentof the user. It is noted that a large number of training images, whichinclude a substantial number and diversity of data related to variousvehicles, results in a robustly trained machine learning model that isconfigured to efficiently and accurately identify the dimensions (andother characteristics) of a vehicle. In embodiments, the machinelearning model may be trained using machine learning algorithms such as,e.g., OpenML or TensorFlow. Other machine learning algorithms withcomparable characteristics are also contemplated.

For example, upon placing the camera 233 in a particular position, avehicle brand and model confirmation page 500, as depicted in FIG. 5 ,may also be displayed on the display 235. By selecting a prompt that isdisplayed, e.g., “confirm” icon, the user may manually classify theexample vehicle 404 according to the vehicle's make and model. In someembodiments, the system described herein may automatically classify theexample vehicle 404 as a 2020 Tacoma TRD, short bed, double cab truckutilizing a plurality of image recognition techniques. These techniquesmay involve analyzing the contours, dimensions, and other such featuresof the example vehicle 404 and performing a comparison of these featureswith one or more databases associated with the server 103, which storesfeatures, dimensions, and contours of various vehicles in one or moredatabases. One or more machine learning techniques may also be utilizedas part of the image recognition techniques to accurately andefficiently classify the example vehicle 404.

FIG. 6 depicts an example page in the display 235 in which aregistration feature 600 such as virtual overlay of a grille isdisplayed, according to one or more embodiments described andillustrated herein. As illustrated in FIG. 6 , the registration feature600 may appear as overlaying or superimposed over portions of theexample vehicle 404 located in the physical environment of the user.Specifically, the registration feature 600 may appear as overlaying alive video stream (or an image) of the example vehicle 404 locatedwithin the physical environment of the user. In embodiments, thesimultaneous display of both the registration feature 600 in the virtualenvironment and the live video stream of the example vehicle 404 in thephysical environment on the display 235 may be the augmented realityenvironment of the augmented reality system described herein. Withinthis augmented reality environment, a user may perform various actionsrelative to example vehicle 404, as depicted and described withreference to FIGS. 8-11 .

The registration feature 600 may vary in shape, size, and dimensions,depending on the type and brand of a vehicle. The page on which theregistration feature 600 appears may also include instructions for theuser to perform certain actions with respect to the example vehicle 404,such as positioning the registration feature 600 near a front portion ofthe example vehicle 404. The front portion of the example vehicle 404may include a grille shaped pattern that is similar to the registrationfeature 600. In embodiments, as depicted in FIG. 6 , an instruction mayread “Scan Grille Located In Front of the Truck”.

FIG. 7 depicts a user positioning the camera 233 of the electronicdevice 104 in compliance with instructions displayed on a page of thedisplay 235, according to one or more embodiments described andillustrated herein. Specifically, FIG. 7 depicts a user moving theelectronic device 104 to align the registration feature 600 (e.g., avirtual image of the grille) with the grille shaped pattern 702 formedon the front of the example vehicle 404, e.g., based on instructionssuch as “Scan Grille Located In Front of the Truck”. In embodiments, theregistration feature 600 may remain stationary within an augmentedreality environment on a page displayed on the user interface 300, whileone or more objects appearing in the background of the display 235(e.g., real world objects in the live video stream) may vary, e.g.,based on movement of the electronic device 104. In embodiments, the usermay move the electronic device 104 near a portion of the example vehicle404 such that the registration feature 600 aligns with correspondingfeatures such as the grille shaped pattern 702, as depicted in FIG. 7 .

Upon alignment, the augmented reality system described herein mayretrieve a virtual wireframe model within an augmented realityenvironment. In some embodiments, the virtual wireframe model may begenerated by combining multiple wireframe model segments, which may begenerated by analyzing perspective images of the example vehicle 404 inthe physical environment. These perspective images may correspond withdifferent orientations of the vehicle in the physical environment.Details regarding the manner in which the virtual wireframe model isgenerated and utilized will be described later on in this disclosure.Furthermore, the process of aligning a fixed sized registration featurewith the corresponding feature on the vehicle enables the electronicdevice 104 to determine the dimensionality of the image data beingreceived by the camera 233. The dimensionality may be continuallyupdated through the motion of the electronic device 104 based on inputfrom the one or more motion sensors 234 of the electronic device 104. Inother words, the electronic device 104 is able to determine with a highdegree of accuracy the real-world dimensions of the object within theimage data based the alignment of the registration feature 600 which hasknown dimensions with the corresponding feature on the object. In thepresent example, this is the grille of the vehicle.

Once the electronic device 104 ascertains dimensionality data for theimage data, the electronic device 104 may register a multi-dimensionalwireframe model of the identified object with the real-world objectbeing captured by the camera 233. For example, FIG. 8 depicts anillustrative example of a multi-dimensional wireframe model of theidentified object being registered (i.e., correctly aligned) with theidentified object within the real-world environment as captured by theimaging device, according to one or more embodiments described andillustrated herein. It should be understood that the multi-dimensionalwireframe model may not be visible to the user on the display 235, butrather be hidden from view so that the user merely sees a real-time viewof the vehicle as the user is instructed to move about the environmentto capture additional perspectives of the object so that the electronicdevice 104 may accurately lock the multidimensional model to thereal-world object.

In response to capturing various perspective images (a first perspectiveimage) of the example vehicle 404, the processor 230 may identifyadditional features of the vehicle such as the front bumper, front headlights, front hood, and the back portions of the side view mirrors.These additional features may be correlated with the correspondingportions of the multi-dimensional wireframe model so that when virtualaccessories are selected to be added to the vehicle in the augmentedreality environment, they may be attached to the multi-dimensionalwireframe model and continuous processing of the real-time image datacan be avoided in order to accurately and seamlessly position thevirtual accessory in the augmented reality environment with respect toits installation location on the vehicle. Other characteristics such ascontours of the vehicle and the presence of various components on thevehicle, may also be identified by the processor 230 using the trainedmachine learning model. Thereafter, using these dimensions, theprocessor 230 may associate the multi-dimensional wireframe model 800with the real-world representation of the object. As depicted in FIG. 8, the multi-dimensional wireframe model 800 appears to overlay the frontbumper, front head lights, front hood, and the back portions of theexample vehicle 404. These portions are located in the first perspectiveimage of the example vehicle 404. Corresponding portions of themulti-dimensional wireframe model 800 may mapped onto the contours ofthe example vehicle 404.

In some embodiments, the multi-dimensional wireframe model 800 may notbe visible to a user, but nonetheless overlay the front portions of thevehicle, as described above. Additionally, in some embodiments, agraphical user interface may be displayed having manual controls foradjusting the multi-dimensional wireframe model 800. The graphical userinterface may include icons such as “rotate” and “shift”. When selected,these icons enable a user to modify the position of themulti-dimensional wireframe model 800 relative to the example vehicle404 within the augmented reality environment. The page displayed in thedisplay 235 may also include instructions for the user to align thecamera 233 to capture image data of another portion of the vehicle. Forexample, instructions may be for the user to align the camera 233 tocapture image data of the side mirror adjacent to the driver seat.

FIG. 9 depicts another portion of the multi-dimensional wireframe model800 overlaying the driver side portions of the example vehicle 404,according to one or more embodiments described and illustrated herein.Specifically, in compliance with the above described instructions, theuser may align the camera 233 with the side mirror adjacent to thedriver seat. Thereafter, another perspective image (e.g., a secondperspective image) corresponding to another orientation of the examplevehicle 404 in the physical environment of the user may be captured. Forexample, the captured image may include areas ranging from the frontright headlight to the back right-tire of the vehicle and areas from thebottom of the front and back right tires to the top portion of theexample vehicle 404 on the driver side. Thereafter, the processor 230may identify dimensions of the example vehicle 404 specific to theseareas. Using these dimensions, the processor 230 may associate thisadditional portion of the multi-dimensional wireframe model 800 with theimage data of the real-world environment. The multi-dimensionalwireframe model 800 may overlay and map to corresponding areas of theexample vehicle 404 included in the second perspective image, e.g.,portions from the bottom of the front and back tires to the top portionon the driver side of the example vehicle 404.

FIG. 10 depicts another illustrative view of the multi-dimensionalwireframe model 800 being registered (i.e., correctly aligned) with theidentified object within the real-world environment as additional pointsof view of the identified object are captured by the imaging device ofthe device, according to one or more embodiments described andillustrated herein. Specifically, in compliance with certaininstructions displayed on the display 235, the user may align the camera233 with the side mirror adjacent to the passenger seat. Thereafter,another image (e.g., a third perspective image) corresponding to anotherorientation of the example vehicle 404 may be captured, namely an imagethat includes the areas ranging from the front left headlight to theback-left tire and areas from the bottom of these tires to the topportion on the passenger side of the example vehicle 404. Thereafter,the processor 230 may, using the machine learning trained modelaccessible via the software application, identify dimensions of theexample vehicle 404 specific to these areas. Using these dimensions, theprocessor 230 may map the corresponding portions of themulti-dimensional wireframe model 800 to the real-world vehicle capturedin the image data. That is, the multi-dimensional wireframe model 800may be a three-dimensional spatial model that is mapped onto thecontours of the vehicle corresponding to the passenger side portions ofthe example vehicle 404.

Once the multi-dimensional wireframe model 800 is registered (i.e.,correctly aligned) to the real-world vehicle captured in the image dataas depicted in FIG. 10 , the electronic device 104 is ready to beginattaching virtual accessories to the multi-dimensional wireframe modeland display them to the user in the augmented reality environmentpresented on the display 235. FIG. 11 depicts an example virtualaccessory corresponding to the identified object being presented as aseamless component added to the identified object within the real-worldenvironment on the display of the device, according to one or moreembodiments described and illustrated herein. For example, the systemdescribed herein may display a vehicle add-on features page so the usermay select one or more accessories to be installed virtually on thevehicle. The display 235 may include a plurality of symbolscorresponding to various vehicle components, e.g., rims, running boards,etc. Each of these symbols may be selectable. In embodiments, uponselection of a symbol corresponding to a virtual component (e.g., avehicle component) from the vehicle add-on features page on the display235, the system described herein may attach the virtual component to thecorresponding portion of the multi-dimensional wireframe model 800. Itis noted that the multi-dimensional wireframe model 800 may overlay allportions of a live video stream of the example vehicle 404 in thephysical environment of the user, and both the multi-dimensionalwireframe model 800 and the live video stream of the example vehicle 404may simultaneously be displayed on the display 235 (i.e. the augmentedreality environment). In this way, the virtual accessory may also bedisplayed within the augmented reality environment.

Still referring to FIG. 11 , symbols 1102 and 1104 associated a roofrack and a running board may be selected. In response, the processor 230may, automatically and without user intervention, attach (or instructthe software application to attach) virtual components associated withthe roof rack and the running board onto a portion of themulti-dimensional wireframe model 800. Thereafter, these virtualcomponents may appear to overlay an area on the roof of the examplevehicle 404 and the area in between the front right and back right tiresof the vehicle within the augmented reality environment. Additionally,as illustrated, these components are displayed on portions of thevehicle such that the virtual components appear realistic andwell-integrated with the example vehicle 404 in the augmented realityenvironment. Primarily because the virtual components correspond tospecific portions of the multi-dimensional wireframe model 800, which inturn has been generated, using the trained machine learning model, so asto accurately fit the contours and dimensions of the example vehicle404. In this way, any virtual components associated with various vehicleadd-on features appear as if these components were real world objectsinstalled on the example vehicle 404 within the physical environment ofthe user.

In some embodiments, a user may select a virtual accessory associatedwith a vehicle (e.g., a vehicle add-on accessory), a version of whichmay already be installed on the example vehicle 404. For instance, theexample vehicle 404 may include a running board, and the user may selecta running board of a different brand with different dimensions. Inresponse, the system described herein may attach the selected virtualaccessory (e.g., the virtual running board) onto the multi-dimensionalwireframe model 800 such that the selected virtual running board may bedisplayed over a current running board within the augmented realityenvironment. To provide a user with a realistic view of the manner inwhich the selected running board may appear on the example vehicle 404,e.g., once it is installed on the vehicle, the system may utilize one ormore occlusions (e.g., digital occlusions). These occlusions may begenerated near locations on the multi-dimensional wireframe model inwhich the virtual component is attached and serve the purpose ofpreventing aspects of the currently installed running board from beingdisplayed. In this way, the system described herein presents users witha display of a virtual component within an augmented reality environmentsuch that the user may perceive the virtual component as accuratelyinstalled on a real-world object, namely the example vehicle 404 withinthe physical environment of the user.

FIG. 12 schematically depicts an example quote for the object and theselected accessories depicted in the augmented reality environmentpresented on the display of the device, according to one or moreembodiments described and illustrated herein. For example, after a userselects one or more virtual accessories, and these virtual accessoriesare displayed accurately and realistically on various parts of theexample vehicle 404, the system described herein may display aconfiguration page that lists various types of information related tothe example vehicle 404. For example, the configuration page may listthe year, make, and model of the example vehicle 404, various vehicleadd-features (e.g., virtual components associated with these featuresmay have been displayed as overlaying the example vehicle 404 in theaugmented reality environment), and the total cost or purchase price(e.g., pricing information) associated with the example vehicle 404.Additionally, the user may be provided with an option to save thevehicle and the selected vehicle add-on features. In some embodiments,images of the example vehicle 404 with the one or more virtualcomponents included on these images may be saved.

FIG. 13 depicts an illustrative flow diagram 1300 of a method forfacilitating an accurate, precise, and seamless presentation of avirtual accessory added to the identified object within the real-worldenvironment on the display of the device in the augmented realityenvironment, according to one or more embodiments described andillustrated herein. At block 1310, the processor 230 may generate, usinga trained machine learning model, the multi-dimensional wireframe model800 within an augmented reality environment. The multi-dimensionalwireframe model 800 may correspond to dimensions of a vehicle in aphysical environment of the user. In some embodiments, the processor 230may use the trained machine learning model to analyze the subject matterof perspective images of the example vehicle 404 that may be capturedfrom different orientations. These images may be analyzed, using thetrained machine learning model, to identify various dimensions of thevehicle. Thereafter, the processor 230 may (or instruct the softwareapplication to) map three-dimensional spatial models onto one or moreareas of the vehicle included in the perspective images. In this way,the multi-dimensional wireframe model 800 corresponding to theidentified object may be retrieved.

At block 1320, the processor 230 may enable a user to select a virtualcomponent by selecting a symbol associated with the virtual componentthat is included on a page displayed on the display 235 of theelectronic device 104. In embodiments, the user input may be based on auser clicking the symbol via an external device (e.g., a mouse), or bychoosing the symbol using the user's fingers. At block 1330, in responseto the selection, the processor 230 may attach the virtual accessory toa portion of the multi-dimensional wireframe model 800 within theaugmented reality environment. Thereafter, at block 1340, the processor230 may display, within the augmented reality environment, the virtualcomponent as overlaying a part the vehicle within at least one of afirst perspective image and a second perspective image of the vehicle inthe physical environment. It is noted that the part of the vehicle maycorrespond to the portion of the multi-dimensional wireframe model 800on which the virtual component is attached.

In yet other alternative illustrative embodiments of the presentdisclosure, the virtual automotive accessories may be customized andlocked onto the real-world vehicles in the physical environment. FIG. 14is a block diagram of a system 1400 and flow diagram for thecollaborative design and customization of various accessories ofvehicles. System 1400 may be used in conjunction with any of thecomputing networks described herein, such as the computing network ordevices of FIG. 1 or 2 . In addition, system 1400 may include a database1402 that includes data related to the vehicles available forcustomization and the accessories available for customization. Anotherdatabase 1404 is included that stores data related to baselineautomotive accessories, which are basic, generalized accessory designsthat may be modified in a variety of ways, as will be described below.

Database 1406 includes data related to color and designattributes/characteristics of the baseline accessories that can becustomized. In certain embodiments, these characteristics may includethe shape, material used, color, logo, texture, angle, slope ordimensions of the baseline accessories. Database 1408 includes thesoftware and data necessary to achieve the wireframe modeling describedherein to optimize the attachment of the virtual accessories to thephysical vehicle in the augmented reality environment. Database 1410stores data related to the customized designs of various users andprovides the data retrieval for the aggregation functionality describedherein.

Still referring to FIG. 14 , database 1402 is used in conjunction withthe selection of the accessory desired to be customized at block 1412.Database 1402 will include all accessories available for sale. Inaddition, although not shown, database 1402 may be in communication withdatabase 1410 in order to retrieve recommended accessory designs and/orother accessories which have previously been customized by other users.Such accessories and/or their customized designs may be recommended tothe users at block 1414.

Databases 1404 and 1406 work in conjunction with block 1416 wherecertain elements of the selected accessory (e.g., a baseline accessoryor a pre-customized recommended accessory) is selected forcustomization. For example, that element (or characteristic) may be thecolor of the accessory. In other examples, it may the texture of thematerial used to make the accessory, etc. In yet other examples, theelement may be the slope of a surface or an angle of the accessory. Atblock 1418, the customization is performed via, for example, a userdisplay of the interactive system 1400.

At block 1420, system 1400 then attaches the customized accessory to thephysical vehicle in the augmented reality environment via, for example,the wireframe modeling described herein. The customized accessory canthen be viewed on the actual vehicle in the augmented realityenvironment, at block 1422 via the user device. At block 1424, system1400 may then transmit the finalized accessory to a design or salesdepartment of the vehicle manufacturer, where the information is storedin database 1410. In certain examples, the finalized designs may beaggregated from multiple users and used to determine which designs areworthy of manufacturing, hence the collaborative aspect of the presentdisclosure. For example, if a certain threshold number of accessoriesare sought to be customized, the system 1400 may determine thataccessory is worthy of being offered for customization (e.g., based onprofitability). In other examples, user customizations may berecommended to other users based upon their profiles or relatedcharacteristics. Thereafter, system 1400 may communicate the decision(yes or no) to the user via a user device to inform the user if theirselected accessory can be customized or not.

FIG. 15 is a flow chart of a method 1500 for customizing and displayingaccessories in an augmented reality, according to certain illustrativemethods of the present invention. At block 1502, the system firstobtains a selected automotive accessory that corresponds to a physical(real-world) vehicle. For example, the user may be on site at a vehiclesales lot and viewing a physical vehicle on the lot through a viewinglens of the user's device. The user may then select, via his customerdevice through an application, an accessory available for customizationthat corresponds to that vehicle.

At block 1504, the system then interactively allows the user tocustomize the automotive accessory as described herein. In certainexamples, the color, shape, material, dimensions, texture, angle orslope of various elements of the accessory may be altered and customizedfor the user. For example, in certain embodiments, system 1400 displays(on the user device) a baseline generic accessory (e.g., grille). Thegrille will have a number of elements which can be customized such as,for example, the dimension, material, etc. of the grille.

In yet other examples, system 1400 may recommend other customizeddesigns selected by other use. In other examples, system 1400 mayanalyze the user's profile to find related profiles of other users andrecommend designs selected by those related users. Once thecustomization has concluded, the system then displays (via a display ofthe user device) the customized automotive accessory overlaid on thephysical vehicle.

With reference to FIGS. 14 and 15 , thereafter, the user may save orotherwise enter the finalized customized accessory. The system 1400 maythen evaluate the accessory to determine if it can be produced. Suchanalysis may include transmitting the finalized design to a sales ordesign team. In other examples, the system 1400 may compare the designto other designs requested by other users to determine if there issufficient demand to justify production of the accessory. Here, system1400 may aggregate the number of related or same customized accessoriesrequested by other users and, based upon the aggregate number, determineif the accessory is cost efficient to produce. In other embodiments, athreshold number of customer requests must have been received in orderto approve the user's customized design.

Furthermore, any of the illustrative methods described herein may beimplemented by a system comprising processing circuitry or anon-transitory computer readable medium comprising instructions which,when executed by at least one processor, causes the processor to performany of the methods described herein.

Although various embodiments and methods have been shown and described,the disclosure is not limited to such embodiments and methods and willbe understood to include all modifications and variations as would beapparent to one skilled in the art. Therefore, it should be understoodthat the disclosure is not intended to be limited to the particularforms disclosed. Rather, the intention is to cover all modifications,equivalents and alternatives falling within the spirit and scope of thedisclosure as defined by the appended claims.

What is claimed is:
 1. A computer-implemented method for displayingvehicle accessories in an augmented reality environment, the methodcomprising: generating an image of a real-world physical vehicle in anaugmented reality environment; obtaining a selection of a virtualautomotive accessory corresponding to the physical vehicle; in responseto user input, customizing the automotive accessory for the physicalvehicle; and displaying the customized automotive accessory to the userin an augmented reality environment such that the customized automotiveaccessory is overlaid atop the physical vehicle, wherein customizing theautomotive accessory comprises recommending customizations to the userbased upon profiles of other users.
 2. The computer-implemented methodas defined in claim 1, further comprising determining whether thecustomized automotive accessory can be produced.
 3. Thecomputer-implemented method as defined in claim 1, further comprising:aggregating customized automotive accessories from multiple users; andbased upon the aggregate customized automotive accessories, determiningwhich customized automotive accessories can be produced.
 4. Thecomputer-implemented method as defined in claim 1, wherein customizingthe automotive accessory comprises: displaying a baseline automotiveaccessory to the user via a user device, wherein the baseline automotiveaccessory has characteristics which can be adjusted within predeterminedtolerances; obtaining, via the user device, characteristics selected bythe user; and based upon the selected characteristics, generating thecustomized automotive accessory.
 5. The computer-implemented method asdefined in claim 4, wherein the characteristics can include one or moreof a shape, material, color, texture, angle, slope or dimension of thebaseline automotive accessory.
 6. The computer-implemented method asdefined in claim 1, wherein the augmented reality environment is viewedthrough a viewing lens of the user device.
 7. A system for displayingvehicle accessories in an augmented reality environment, the systemcomprising: a processor to perform operations comprising: generating animage of a real-world physical vehicle in an augmented realityenvironment; obtaining a selection of a virtual automotive accessorycorresponding to the physical vehicle; in response to user input,customizing the automotive accessory for the physical vehicle; anddisplaying the customized automotive accessory to the user in anaugmented reality environment such that the customized automotiveaccessory is overlaid atop the physical vehicle, wherein customizing theautomotive accessory comprises recommending customizations to the userbased upon profiles of other users.
 8. The system as defined in claim 7,further comprising determining whether the customized automotiveaccessory can be produced.
 9. The system as defined in claim 7, furthercomprising: aggregating customized automotive accessories from multipleusers; and based upon the aggregate customized automotive accessories,determining which customized automotive accessories can be massproduced.
 10. The system as defined in claim 7, wherein customizing theautomotive accessory comprises: displaying a baseline automotiveaccessory to the user via a user device, wherein the baseline automotiveaccessory has characteristics which can be adjusted within predeterminedtolerances; obtaining, via the user device, characteristics selected bythe user; and based upon the selected characteristics, generating thecustomized automotive accessory.
 11. The system as defined in claim 10,wherein the characteristics can include one or more of a shape,material, color, texture, angle, slope or dimension of the baselineautomotive accessory.
 12. The system as defined in claim 7, wherein theaugmented reality environment is viewed through a viewing lens of theuser device.
 13. A method for displaying vehicle accessories in anaugmented reality environment, the method comprising: obtaining, from auser via a user device, a customized automotive accessory correspondingto a real-world physical vehicle; and displaying the customizedautomotive accessory to the user in an augmented reality environmentsuch that the customized automotive accessory is overlaid atop thephysical vehicle, wherein customizing the automotive accessory comprisesrecommending customizations to the user based upon profiles of otherusers.
 14. The method as defined in claim 13, wherein obtaining thecustomized automotive accessory comprises: uploading a pre-customizedautomotive accessory to the user device; or in response to user input,customizing the automotive accessory using selections of the userinputted via the user device.
 15. The method as defined in claim 13,further comprising: determining whether the customized automotiveaccessory can be produced; or aggregating customized automotiveaccessories from multiple users, and based upon the aggregate customizedautomotive accessories, determining which customized automotiveaccessories can be produced.
 16. The method as defined in claim 13,wherein customizing the automotive accessory comprises: displaying abaseline automotive accessory to the user via a user device, wherein thebaseline automotive accessory has characteristics which can be adjustedwithin tolerances; obtaining, via the user device, characteristicsselected by the user; and based upon the selected characteristics,generating the customized automotive accessory.
 17. The method asdefined in claim 13, wherein customizing the automotive accessorycomprises recommending customizations to the user.
 18. The method asdefined in claim 16, wherein the augmented reality environment is viewedthrough a viewing lens of the user device.